Capture app context in analytics for every command run inside an app by nelsonwittwer · Pull Request #7771 · Shopify/cli

nelsonwittwer · 2026-06-09T19:27:40Z

What

Makes command analytics capture app context for every CLI command run inside an app — not just commands that explicitly load an app.

Why

App-level telemetry is already flowing, but only as a side effect of commands that authenticate and load an app.

Commands that don't load an app (search, fetch-doc, version, upgrade, …) emit nothing, so we can't tell what app context they were run in. We need this signal everywhere so we can judge the effectiveness of commands, especially when run by agents in different harnesses/models.

How

The public_command_metadata hook runs on the postrun of every command. It now calls a best-effort helper, logAppContextMetadataIfAuthenticated(cwd()), which — only when the user is already authenticated and is inside an app project — loads the app from the current directory and broadcasts the same app-context block a shopify app * command would (api_key, project_type, the app_* fields, cmd_app_linked_config_*, and the sensitive app_name).

We do not deduce an app for commands run outside an app directory. The behavior is simply: if the command is run within the path of an app, it broadcasts that app's context; otherwise it broadcasts nothing app-related.

It is strictly best-effort and safe to run everywhere:

Gated on an existing local session (sessionExists()) — never triggers a login, never makes a network request. Anonymous/CI usage stays unenriched.
Short-circuits when api_key is already set — app commands already populated it, so there's no redundant work.
Never prompts — skipPrompts is threaded through localAppContext → getAppConfigurationContext.
Bounded by a 3s timeout — it runs before process exit, so it can't hang the CLI.
Swallows all errors — not an app dir, invalid config, etc. → silently does nothing.

🎩 testing instructions

# 1. Build the branch (run from your CLI repo root)
pnpm nx build app

# 2. Run a non-app command INSIDE an app directory.
#    Replace the cd path with one of your own apps; replace the dev.js path with your checkout.
cd ~/code/luxury-duplication-app
SHOPIFY_CLI_NO_ANALYTICS=1 SHOPIFY_FLAG_VERBOSE=1 node /Users/nelson/src/github.com/Shopify/cli/packages/cli/bin/dev.js version 2>&1 | grep -E '"command"|"api_key"|"project_type"|"app_scopes"|"cmd_app_linked_config_name"'
#   → api_key, project_type, app_scopes, cmd_app_linked_config_name, … all present
#     (the full app-context block, same as a `shopify app *` command)

# 3. Run the SAME command from a directory that is NOT an app (/tmp works on any machine):
cd /tmp
SHOPIFY_CLI_NO_ANALYTICS=1 SHOPIFY_FLAG_VERBOSE=1 node /Users/nelson/src/github.com/Shopify/cli/packages/cli/bin/dev.js version 2>&1 | grep -E '"command"|"api_key"|"project_type"|"app_scopes"'
#   → only "command" — no api_key / project_type / app_* (no app context is broadcast)

…mand Today only commands that load an app (app dev/deploy/...) populate api_key and project_type in Monorail; lightweight commands (search, fetch-doc, version, ...) emit nothing, so there's no way to know what app context they ran in. Enrich the global public_command_metadata hook so that — for users who are ALREADY authenticated and inside an app project — it reads the app from disk and attaches api_key + project_type. It is strictly best-effort: - gated on an existing local session; never triggers a login or any network call - short-circuits when api_key is already set (an app command already loaded the app) - threads skipPrompts through localAppContext so it can never prompt - bounded by a 3s timeout so it can't delay command exit - swallows all errors (not an app dir, invalid config, etc.) Adds a passive, non-interactive `sessionExists()` to @shopify/cli-kit/node/session. No Monorail schema change: api_key/project_type already exist on the event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The public_command_metadata hook now captures the complete app-context analytics block (api_key, project_type, app_*, cmd_app_linked_config_*, cmd_app_all_configs_*, and the sensitive app_name) when an app resolves, persists it as the most-recently-used app, and replays the whole block for commands run outside any app directory. A new public field, cmd_app_context_source, records how the context was resolved: "current_directory" for a real on-disk load, "last_used" when it was replayed from the most-recently-used app. Note: the Monorail backend schema registry must register this field for it to be retained/queryable. The snapshot keeps app identity/shape fields and excludes per-run flags (e.g. cmd_app_warning_*). Enrichment remains best-effort: authenticated users only, no prompts, no network, bounded by a short timeout, all errors swallowed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Reverts the most-recently-used app replay. We no longer deduce an app for commands run outside an app directory. Instead we trust in-app context: any command run within an app path broadcasts the same app-context block a `shopify app *` command would (api_key, project_type, app_*, cmd_app_linked_config_*, and the sensitive app_name), driven by loading the app from the current directory in the public_command_metadata hook. Removes the dedicated MRU LocalStorage store, the cmd_app_context_source Monorail discriminator, and the snapshot/replay helpers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-15T17:16:13Z

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

Some seemingly private modules might be re-exported through public modules.
If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/session.d.ts

@@ -47,6 +47,22 @@ export declare function isUserAccount(account: AccountInfo): account is UserAcco
  * @returns True if the account is a ServiceAccount.
  */
 export declare function isServiceAccount(account: AccountInfo): account is ServiceAccountInfo;
+/**
+ * Reports whether the CLI already has stored credentials, without prompting the
+ * user, opening a browser, or making any network request.
+ *
+ * This is a passive, side-effect-free check: it reads the local session store and
+ * returns  when at least one valid session is present. Unlike the
+ *  functions, it never triggers a login flow and never logs
+ * the user out. Because it does not contact the network, it cannot tell whether the
+ * stored token is currently valid/unexpired — only that credentials exist locally.
+ *
+ * Intended for best-effort, opportunistic behaviour (for example, enriching
+ * telemetry only for users who are already logged in).
+ *
+ * @returns True if local credentials exist, false otherwise.
+ */
+export declare function sessionExists(): Promise<boolean>;
 /**
  * Ensure that we have a valid session with no particular scopes.
  *

github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Jun 9, 2026

nelsonwittwer commented Jun 9, 2026

View reviewed changes

Comment thread packages/app/src/cli/hooks/public_metadata.ts Outdated

Comment thread packages/app/src/cli/services/app-context.ts

Comment thread packages/app/src/cli/services/app-context.ts Outdated

Comment thread packages/app/src/cli/services/app-context.ts Outdated

Address review: drop redundant comments

698c634

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

This was referenced Jun 10, 2026

Deprecate and fix search #7778

Draft

Add shopify doc fetch command to download MD doc from shopify.dev #7766

Draft

Add doc search command (JSON documentation search) #7770

Draft

nelsonwittwer and others added 2 commits June 15, 2026 09:13

nelsonwittwer changed the title ~~Capture app context (api_key/project_type) in analytics for every command~~ Capture app context in analytics for every command run inside an app Jun 15, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture app context in analytics for every command run inside an app#7771

Capture app context in analytics for every command run inside an app#7771
nelsonwittwer wants to merge 4 commits into
mainfrom
telemetry-app-context-everywhere

nelsonwittwer commented Jun 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

nelsonwittwer commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

🎩 testing instructions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 15, 2026

Differences in type declarations

New type declarations

Existing type declarations

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nelsonwittwer commented Jun 9, 2026 •

edited

Loading